{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "690cfdc5",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# Statistics\n",
    ":label:`sec_statistics`\n",
    "\n",
    "Undoubtedly, to be a top deep learning practitioner, the ability to train the state-of-the-art and high accurate models is crucial.  However, it is often unclear when improvements are significant, or only the result of random fluctuations in the training process.  To be able to discuss uncertainty in estimated values, we must learn some statistics.\n",
    "\n",
    "\n",
    "The earliest reference of *statistics* can be traced back to an Arab scholar Al-Kindi in the $9^{\\textrm{th}}$-century, who gave a detailed description of how to use statistics and frequency analysis to decipher encrypted messages. After 800 years, the modern statistics arose from Germany in 1700s, when the researchers focused on the demographic and economic data collection and analysis. Today, statistics is the science subject that concerns the collection, processing, analysis, interpretation and visualization of data. What is more, the core theory of statistics has been widely used in the research within academia, industry, and government.\n",
    "\n",
    "\n",
    "More specifically, statistics can be divided to *descriptive statistics* and *statistical inference*. The former focus on summarizing and illustrating the features of a collection of observed data, which is referred to as a *sample*. The sample is drawn from a *population*, denotes the total set of similar individuals, items, or events of our experiment interests. Contrary to descriptive statistics, *statistical inference* further deduces the characteristics of a population from the given *samples*, based on the assumptions that the sample distribution can replicate the population distribution at some degree.\n",
    "\n",
    "\n",
    "You may wonder: “What is the essential difference between machine learning and statistics?” Fundamentally speaking, statistics focuses on the inference problem. This type of problems includes modeling the relationship between the variables, such as causal inference, and testing the statistically significance of model parameters, such as A/B testing. In contrast, machine learning emphasizes on making accurate predictions, without explicitly programming and understanding each parameter's functionality.\n",
    "\n",
    "\n",
    "In this section, we will introduce three types of statistics inference methods: evaluating and comparing estimators, conducting hypothesis tests, and constructing confidence intervals. These methods can help us infer the characteristics of a given population, i.e., the true parameter $\\theta$. For brevity, we assume that the true parameter $\\theta$ of a given population is a scalar value. It is straightforward to extend to the case where $\\theta$ is a vector or a tensor, thus we omit it in our discussion.\n",
    "\n",
    "\n",
    "\n",
    "## Evaluating and Comparing Estimators\n",
    "\n",
    "In statistics, an *estimator* is a function of given samples used to estimate the true parameter $\\theta$. We will write $\\hat{\\theta}_n = \\hat{f}(x_1, \\ldots, x_n)$ for the estimate of $\\theta$ after observing the samples {$x_1, x_2, \\ldots, x_n$}.\n",
    "\n",
    "We have seen simple examples of estimators before in section :numref:`sec_maximum_likelihood`.  If you have a number of samples from a Bernoulli random variable, then the maximum likelihood estimate for the probability the random variable is one can be obtained by counting the number of ones observed and dividing by the total number of samples.  Similarly, an exercise asked you to show that the maximum likelihood estimate of the mean of a Gaussian given a number of samples is given by the average value of all the samples.  These estimators will almost never give the true value of the parameter, but ideally for a large number of samples the estimate will be close.\n",
    "\n",
    "As an example, we show below the true density of a Gaussian random variable with mean zero and variance one, along with a collection samples from that Gaussian.  We constructed the $y$ coordinate so every point is visible and the relationship to the original density is clearer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1fd6ca33",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:38:54.225972Z",
     "iopub.status.busy": "2023-08-18T19:38:54.225157Z",
     "iopub.status.idle": "2023-08-18T19:38:58.068578Z",
     "shell.execute_reply": "2023-08-18T19:38:58.066767Z"
    },
    "origin_pos": 2,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"246.28125pt\" height=\"198.474375pt\" viewBox=\"0 0 246.28125 198.474375\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T19:38:57.987073</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 198.474375 \n",
       "L 246.28125 198.474375 \n",
       "L 246.28125 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 43.78125 160.918125 \n",
       "L 239.08125 160.918125 \n",
       "L 239.08125 22.318125 \n",
       "L 43.78125 22.318125 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_1\">\n",
       "    <defs>\n",
       "     <path id=\"m089e996f1a\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #1f77b4\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pcf76630914)\">\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"110.814232\" y=\"155.106154\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"196.881804\" y=\"155.106154\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"157.426864\" y=\"155.106154\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"125.239122\" y=\"155.106142\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"167.20008\" y=\"155.094374\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"126.988067\" y=\"151.613557\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"183.222875\" y=\"155.10611\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"120.872164\" y=\"153.372958\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"52.658523\" y=\"155.106154\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"126.959207\" y=\"146.945083\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"138.560894\" y=\"155.103909\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"121.657855\" y=\"147.64906\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"142.674869\" y=\"153.61879\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"121.761375\" y=\"143.288189\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"120.751044\" y=\"140.918785\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"102.634893\" y=\"155.037575\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"132.190934\" y=\"152.948362\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"148.745624\" y=\"154.622236\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"157.515714\" y=\"150.841772\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"150.71139\" y=\"151.196627\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"119.365551\" y=\"140.891221\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"77.584049\" y=\"155.106154\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"131.914149\" y=\"148.462123\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"124.367173\" y=\"135.072888\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"93.958761\" y=\"155.065199\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"62.916358\" y=\"155.099666\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"157.797274\" y=\"146.492482\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"161.34987\" y=\"149.310298\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"98.462215\" y=\"152.45263\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"135.832605\" y=\"148.62878\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.600297\" y=\"151.024208\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"181.769881\" y=\"151.40373\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"158.423906\" y=\"140.369729\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"139.246756\" y=\"146.538805\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"204.288266\" y=\"154.962188\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"113.485486\" y=\"149.774101\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"137.616308\" y=\"141.927985\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"163.584504\" y=\"147.920338\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"181.026991\" y=\"147.897321\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"123.912314\" y=\"130.17295\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"185.914626\" y=\"149.971533\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"153.683025\" y=\"145.56264\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"139.861187\" y=\"139.718282\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"138.444805\" y=\"134.207183\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"184.935384\" y=\"143.689601\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"132.755838\" y=\"140.676569\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"141.365446\" y=\"136.686915\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"130.956591\" y=\"137.936075\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.324223\" y=\"145.49709\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"135.459626\" y=\"131.16744\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"104.024667\" y=\"149.092709\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"159.613412\" y=\"135.703539\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"168.788448\" y=\"150.525135\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.982197\" y=\"140.147203\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"181.563581\" y=\"139.791146\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"67.016735\" y=\"153.604164\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"133.887761\" y=\"128.977595\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"112.279551\" y=\"138.935579\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"176.629341\" y=\"151.56884\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"73.97428\" y=\"152.99943\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"139.611424\" y=\"125.427306\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"119.436233\" y=\"133.557478\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"157.629654\" y=\"130.952907\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"99.970537\" y=\"146.683382\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"132.798079\" y=\"126.063092\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"150.427112\" y=\"144.20281\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"169.113419\" y=\"146.650138\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"124.460369\" y=\"125.520713\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"128.720013\" y=\"130.036325\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"200.164475\" y=\"151.454383\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"202.037166\" y=\"147.801301\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"148.965011\" y=\"141.886217\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"197.386287\" y=\"146.994424\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"125.121769\" y=\"120.970776\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"132.163371\" y=\"120.885143\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"128.460555\" y=\"121.879728\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"135.750511\" y=\"116.702829\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"153.946708\" y=\"135.597096\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"172.505686\" y=\"148.869636\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.260587\" y=\"134.072076\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"116.405989\" y=\"140.289792\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"94.251519\" y=\"148.866214\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"170.804892\" y=\"142.139837\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"143.320815\" y=\"137.722826\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"122.944919\" y=\"114.806427\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"144.767816\" y=\"138.531561\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"106.822084\" y=\"137.540364\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"100.427146\" y=\"141.419459\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"160.915359\" y=\"132.058885\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"83.292936\" y=\"154.512685\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"91.143525\" y=\"149.883567\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"93.308651\" y=\"142.532226\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"120.749872\" y=\"117.016585\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"139.020097\" y=\"115.209853\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"114.401784\" y=\"135.359684\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"135.490218\" y=\"110.259259\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"108.21631\" y=\"128.516332\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"147.167495\" y=\"136.367308\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"146.843528\" y=\"132.263138\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"140.015191\" y=\"111.979302\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"131.341983\" y=\"112.535701\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"116.345028\" y=\"131.176018\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"150.882441\" y=\"129.996327\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"174.50229\" y=\"144.656003\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"157.533411\" y=\"122.355405\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"136.060337\" y=\"103.200738\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"130.892254\" y=\"108.671981\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"95.020497\" y=\"138.102691\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"132.145778\" y=\"100.981737\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"116.778555\" y=\"126.133977\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"138.979347\" y=\"101.891757\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"134.493986\" y=\"94.010314\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"130.502977\" y=\"100.491837\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"134.277613\" y=\"88.088574\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"179.289294\" y=\"139.083675\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"151.999434\" y=\"126.637806\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"86.696361\" y=\"151.130895\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.464607\" y=\"121.306467\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"136.354483\" y=\"86.922742\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"149.178062\" y=\"122.006649\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"117.663484\" y=\"119.550354\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"174.37147\" y=\"139.329303\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"149.040064\" y=\"117.938908\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"102.758031\" y=\"136.304566\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"137.902523\" y=\"87.415544\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"134.88689\" y=\"78.293149\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"164.939752\" y=\"139.193474\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"163.236012\" y=\"132.212354\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"147.470874\" y=\"117.276893\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"113.362409\" y=\"120.809971\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"129.631867\" y=\"98.27816\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"138.321726\" y=\"82.944092\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"115.58007\" y=\"117.122756\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"184.056687\" y=\"134.097677\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"152.722098\" y=\"118.425962\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"99.216273\" y=\"134.22486\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"128.583682\" y=\"99.965708\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"142.237313\" y=\"110.164153\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"119.786232\" y=\"107.496795\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"100.672942\" y=\"129.257668\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"171.527837\" y=\"133.67049\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"114.147674\" y=\"112.926886\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"82.950742\" y=\"148.319158\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"194.849944\" y=\"148.019605\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"104.86544\" y=\"128.391256\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"103.427639\" y=\"124.111438\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"161.2877\" y=\"122.228104\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"83.222868\" y=\"143.967459\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"115.457907\" y=\"107.861485\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"136.789754\" y=\"71.888235\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"130.610564\" y=\"84.08125\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"71.618088\" y=\"150.450639\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.889816\" y=\"110.568666\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"173.122776\" y=\"130.716933\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"107.413142\" y=\"114.044197\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"173.270941\" y=\"126.694075\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"139.296623\" y=\"79.90945\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"143.21326\" y=\"109.991621\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"155.907057\" y=\"116.418969\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"172.402783\" y=\"121.850251\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"111.771457\" y=\"103.007805\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"115.563884\" y=\"101.175884\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"174.506516\" y=\"121.9318\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"129.507182\" y=\"86.693629\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"62.818958\" y=\"149.422199\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"120.449671\" y=\"100.081791\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"176.093549\" y=\"124.392104\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"101.791002\" y=\"119.007869\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"111.482335\" y=\"97.186561\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"144.081503\" y=\"110.086528\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"111.344524\" y=\"93.079568\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"179.905823\" y=\"128.342364\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"154.951928\" y=\"113.277837\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"170.240423\" y=\"119.570533\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"101.244223\" y=\"115.544826\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"163.394934\" y=\"124.634777\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"213.868289\" y=\"155.090522\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"154.057477\" y=\"109.426387\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"122.554077\" y=\"98.115545\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"144.517634\" y=\"107.297129\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"145.302454\" y=\"104.76008\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"88.075695\" y=\"144.221492\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"129.323866\" y=\"83.362174\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"141.147096\" y=\"83.373754\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"140.38666\" y=\"74.343446\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"115.703517\" y=\"93.037363\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"148.403603\" y=\"101.620301\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"138.967195\" y=\"63.133701\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"131.540702\" y=\"68.142157\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"156.276148\" y=\"104.499094\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"141.121885\" y=\"71.572445\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"131.766285\" y=\"63.18655\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"106.0919\" y=\"109.695858\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"125.091843\" y=\"95.903409\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"110.760939\" y=\"87.621186\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"156.76507\" y=\"100.397586\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"101.001473\" y=\"110.948631\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"159.501942\" y=\"103.474952\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"84.74984\" y=\"138.21195\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"100.277699\" y=\"109.26654\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"142.052968\" y=\"74.665275\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.29663\" y=\"88.12492\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"167.262851\" y=\"125.179464\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"149.65778\" y=\"97.644335\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"143.189239\" y=\"79.107557\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"151.027291\" y=\"95.204252\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"107.730599\" y=\"92.560002\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"111.214753\" y=\"77.39119\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"134.785125\" y=\"55.761587\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"151.633054\" y=\"92.058082\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"130.011188\" y=\"65.934089\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"69.042412\" y=\"147.231205\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"147.223523\" y=\"88.46747\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"191.771116\" y=\"150.319247\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"136.618025\" y=\"49.578676\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"64.012199\" y=\"143.890593\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"122.939619\" y=\"90.605587\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"128.798254\" y=\"70.581012\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"148.424592\" y=\"83.239321\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"139.475842\" y=\"48.857857\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"122.917091\" y=\"85.874425\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"72.112787\" y=\"143.541515\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"103.310439\" y=\"100.934742\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"116.20873\" y=\"87.02758\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"135.69997\" y=\"44.224192\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"89.513391\" y=\"138.372119\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"172.019923\" y=\"108.705941\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"94.634603\" y=\"129.390157\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"123.976709\" y=\"82.150226\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"139.532242\" y=\"43.161824\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"163.273771\" y=\"116.117619\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"127.173412\" y=\"74.574223\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"146.126334\" y=\"80.690588\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"136.45918\" y=\"36.502705\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"98.37003\" y=\"112.908045\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"145.585984\" y=\"75.983521\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"150.851517\" y=\"79.993628\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"142.840545\" y=\"60.154648\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"135.378315\" y=\"34.942376\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"123.839767\" y=\"75.700155\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"180.697067\" y=\"123.74165\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"109.935569\" y=\"73.862409\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"144.385508\" y=\"64.728935\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"94.785456\" y=\"123.062098\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"131.492148\" y=\"44.315631\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"186.283442\" y=\"136.731592\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"214.070074\" y=\"150.888155\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"116.206916\" y=\"82.21132\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"169.310305\" y=\"112.525605\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"112.791627\" y=\"69.846331\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"99.353937\" y=\"101.81991\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"129.029705\" y=\"55.354149\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"132.543156\" y=\"34.218675\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"97.929145\" y=\"105.218737\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"131.557303\" y=\"32.995699\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"96.30772\" y=\"109.218038\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"143.046728\" y=\"53.446601\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"179.28729\" y=\"120.148887\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"135.731586\" y=\"24.151549\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"125.735526\" y=\"67.828597\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"169.927747\" y=\"106.019699\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"115.235655\" y=\"72.890412\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"76.435978\" y=\"144.949733\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"149.544513\" y=\"70.270371\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"106.862125\" y=\"86.990157\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"121.263556\" y=\"73.290027\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"164.611864\" y=\"114.386133\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"160.25557\" y=\"98.679447\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"158.137988\" y=\"89.582244\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"95.263854\" y=\"109.625414\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"160.533568\" y=\"92.404034\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"186.01381\" y=\"130.976487\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"180.919521\" y=\"114.327851\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"143.76751\" y=\"52.867425\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"171.344193\" y=\"98.199118\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"98.654464\" y=\"92.015135\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"143.888047\" y=\"49.282833\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"126.866229\" y=\"58.689085\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"143.93538\" y=\"45.312016\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"159.332469\" y=\"84.101835\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"159.136865\" y=\"79.442005\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"217.558111\" y=\"151.288929\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"118.821522\" y=\"74.930598\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"166.44926\" y=\"108.6744\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"129.874323\" y=\"34.63027\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"230.403615\" y=\"155.10599\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"107.97178\" y=\"74.981245\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"175.451195\" y=\"107.592803\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"144.846641\" y=\"45.831304\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"126.690716\" y=\"52.907404\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"140.795895\" y=\"25.621497\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"120.823162\" y=\"66.208615\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"132.862104\" y=\"19.424573\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"118.528739\" y=\"67.747173\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"122.325217\" y=\"56.12553\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"156.86565\" y=\"77.171757\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"108.11838\" y=\"69.723189\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"134.246249\" y=\"15.254956\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"165.676762\" y=\"104.196646\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "     <use xlink:href=\"#m089e996f1a\" x=\"149.137121\" y=\"61.091985\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 78.19522 160.918125 \n",
       "L 78.19522 22.318125 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m972e027ed6\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m972e027ed6\" x=\"78.19522\" y=\"160.918125\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(70.824126 175.516563) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 135.192155 160.918125 \n",
       "L 135.192155 22.318125 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m972e027ed6\" x=\"135.192155\" y=\"160.918125\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(132.010905 175.516563) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 192.189091 160.918125 \n",
       "L 192.189091 22.318125 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m972e027ed6\" x=\"192.189091\" y=\"160.918125\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(189.007841 175.516563) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(138.471875 189.194688) scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 43.78125 155.106154 \n",
       "L 239.08125 155.106154 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <defs>\n",
       "       <path id=\"m18f88919f8\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m18f88919f8\" x=\"43.78125\" y=\"155.106154\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(20.878125 158.905373) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 43.78125 123.400064 \n",
       "L 239.08125 123.400064 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m18f88919f8\" x=\"43.78125\" y=\"123.400064\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 0.1 -->\n",
       "      <g transform=\"translate(20.878125 127.199283) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 43.78125 91.693974 \n",
       "L 239.08125 91.693974 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m18f88919f8\" x=\"43.78125\" y=\"91.693974\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(20.878125 95.493193) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 43.78125 59.987884 \n",
       "L 239.08125 59.987884 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m18f88919f8\" x=\"43.78125\" y=\"59.987884\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.3 -->\n",
       "      <g transform=\"translate(20.878125 63.787103) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 43.78125 28.281794 \n",
       "L 239.08125 28.281794 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m18f88919f8\" x=\"43.78125\" y=\"28.281794\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(20.878125 32.081013) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- density -->\n",
       "     <g transform=\"translate(14.798438 109.950938) rotate(-90) scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-64\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"188.378906\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"240.478516\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"268.261719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"307.470703\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 52.658523 153.197235 \n",
       "L 56.078336 152.423342 \n",
       "L 59.213171 151.487204 \n",
       "L 62.063015 150.405081 \n",
       "L 64.627881 149.207715 \n",
       "L 67.192741 147.765089 \n",
       "L 69.757601 146.043332 \n",
       "L 72.037483 144.25071 \n",
       "L 74.317358 142.186466 \n",
       "L 76.597233 139.827788 \n",
       "L 78.877115 137.153749 \n",
       "L 81.15699 134.146283 \n",
       "L 83.436866 130.791107 \n",
       "L 86.001729 126.589398 \n",
       "L 88.566592 121.931424 \n",
       "L 91.131455 116.823953 \n",
       "L 93.981302 110.646046 \n",
       "L 97.116131 103.294752 \n",
       "L 100.820931 93.98551 \n",
       "L 105.950658 80.386519 \n",
       "L 113.075274 61.507974 \n",
       "L 116.495091 53.109531 \n",
       "L 119.059952 47.34296 \n",
       "L 121.33983 42.710863 \n",
       "L 123.334722 39.10535 \n",
       "L 125.329615 35.969385 \n",
       "L 127.039524 33.688464 \n",
       "L 128.749431 31.808555 \n",
       "L 130.174355 30.562713 \n",
       "L 131.599278 29.618403 \n",
       "L 133.024202 28.982626 \n",
       "L 134.16414 28.699425 \n",
       "L 135.304079 28.618125 \n",
       "L 136.444018 28.739131 \n",
       "L 137.583957 29.061857 \n",
       "L 138.723895 29.584745 \n",
       "L 140.148819 30.515949 \n",
       "L 141.573742 31.749035 \n",
       "L 142.998666 33.274865 \n",
       "L 144.708573 35.476404 \n",
       "L 146.418481 38.060284 \n",
       "L 148.413375 41.522339 \n",
       "L 150.408267 45.420776 \n",
       "L 152.688144 50.343018 \n",
       "L 155.537992 57.072656 \n",
       "L 158.957808 65.767179 \n",
       "L 164.657499 80.988595 \n",
       "L 170.357194 96.027665 \n",
       "L 174.061995 105.206883 \n",
       "L 177.196827 112.417875 \n",
       "L 180.046674 118.451822 \n",
       "L 182.896518 123.946038 \n",
       "L 185.461381 128.412243 \n",
       "L 188.026244 132.422786 \n",
       "L 190.591104 135.986275 \n",
       "L 192.870986 138.79231 \n",
       "L 195.150861 141.275324 \n",
       "L 197.430737 143.455222 \n",
       "L 199.710612 145.354158 \n",
       "L 202.275478 147.18396 \n",
       "L 204.840338 148.722345 \n",
       "L 207.405198 150.003494 \n",
       "L 210.255048 151.165413 \n",
       "L 213.104893 152.093033 \n",
       "L 216.239728 152.889068 \n",
       "L 219.944525 153.587182 \n",
       "L 224.219298 154.144697 \n",
       "L 229.349024 154.567032 \n",
       "L 230.203977 154.618125 \n",
       "L 230.203977 154.618125 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_18\">\n",
       "    <path d=\"M 135.192155 160.918125 \n",
       "L 135.192155 22.318125 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_19\">\n",
       "    <path d=\"M 135.257908 160.918125 \n",
       "L 135.257908 22.318125 \n",
       "\" clip-path=\"url(#pcf76630914)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #800080; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 43.78125 160.918125 \n",
       "L 43.78125 22.318125 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 239.08125 160.918125 \n",
       "L 239.08125 22.318125 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 43.78125 160.918125 \n",
       "L 239.08125 160.918125 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 43.78125 22.318125 \n",
       "L 239.08125 22.318125 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"text_11\">\n",
       "    <!-- sample mean: 0.00 -->\n",
       "    <g transform=\"translate(83.406563 16.318125) scale(0.12 -0.12)\">\n",
       "     <defs>\n",
       "      <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-3a\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 0 \n",
       "L 750 0 \n",
       "L 750 794 \n",
       "z\n",
       "M 750 3309 \n",
       "L 1409 3309 \n",
       "L 1409 2516 \n",
       "L 750 2516 \n",
       "L 750 3309 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "     </defs>\n",
       "     <use xlink:href=\"#DejaVuSans-73\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-61\" x=\"52.099609\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-6d\" x=\"113.378906\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-70\" x=\"210.791016\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-6c\" x=\"274.267578\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-65\" x=\"302.050781\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-20\" x=\"363.574219\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-6d\" x=\"395.361328\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-65\" x=\"492.773438\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-61\" x=\"554.296875\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-6e\" x=\"615.576172\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-3a\" x=\"678.955078\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-20\" x=\"712.646484\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-30\" x=\"744.433594\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-2e\" x=\"808.056641\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-30\" x=\"839.84375\"/>\n",
       "     <use xlink:href=\"#DejaVuSans-30\" x=\"903.466797\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pcf76630914\">\n",
       "   <rect x=\"43.78125\" y=\"22.318125\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from d2l import torch as d2l\n",
    "\n",
    "torch.pi = torch.acos(torch.zeros(1)) * 2  #define pi in torch\n",
    "\n",
    "# Sample datapoints and create y coordinate\n",
    "epsilon = 0.1\n",
    "torch.manual_seed(8675309)\n",
    "xs = torch.randn(size=(300,))\n",
    "\n",
    "ys = torch.tensor(\n",
    "    [torch.sum(torch.exp(-(xs[:i] - xs[i])**2 / (2 * epsilon**2))\\\n",
    "               / torch.sqrt(2*torch.pi*epsilon**2)) / len(xs)\\\n",
    "     for i in range(len(xs))])\n",
    "\n",
    "# Compute true density\n",
    "xd = torch.arange(torch.min(xs), torch.max(xs), 0.01)\n",
    "yd = torch.exp(-xd**2/2) / torch.sqrt(2 * torch.pi)\n",
    "\n",
    "# Plot the results\n",
    "d2l.plot(xd, yd, 'x', 'density')\n",
    "d2l.plt.scatter(xs, ys)\n",
    "d2l.plt.axvline(x=0)\n",
    "d2l.plt.axvline(x=torch.mean(xs), linestyle='--', color='purple')\n",
    "d2l.plt.title(f'sample mean: {float(torch.mean(xs).item()):.2f}')\n",
    "d2l.plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78a3faa3",
   "metadata": {
    "origin_pos": 4
   },
   "source": [
    "There can be many ways to compute an estimator of a parameter $\\hat{\\theta}_n$.  In this section, we introduce three common methods to evaluate and compare estimators: the mean squared error, the standard deviation, and statistical bias.\n",
    "\n",
    "### Mean Squared Error\n",
    "\n",
    "Perhaps the simplest metric used to evaluate estimators is the *mean squared error (MSE)* (or $l_2$ loss) estimator which can be defined as\n",
    "\n",
    "$$\\textrm{MSE} (\\hat{\\theta}_n, \\theta) = E[(\\hat{\\theta}_n - \\theta)^2].$$\n",
    ":eqlabel:`eq_mse_est`\n",
    "\n",
    "This allows us to quantify the average squared deviation from the true value.  MSE is always non-negative. If you have read :numref:`sec_linear_regression`, you will recognize it as the most commonly used regression loss function. As a measure to evaluate an estimator, the closer its value to zero, the closer the estimator is close to the true parameter $\\theta$.\n",
    "\n",
    "\n",
    "### Statistical Bias\n",
    "\n",
    "The MSE provides a natural metric, but we can easily imagine multiple different phenomena that might make it large.  Two fundamentally important are fluctuation in the estimator due to randomness in the dataset, and systematic error in the estimator due to the estimation procedure.\n",
    "\n",
    "\n",
    "First, let's measure the systematic error. For an estimator $\\hat{\\theta}_n$, the mathematical illustration of *statistical bias* can be defined as\n",
    "\n",
    "$$\\textrm{bias}(\\hat{\\theta}_n) = E(\\hat{\\theta}_n - \\theta) = E(\\hat{\\theta}_n) - \\theta.$$\n",
    ":eqlabel:`eq_bias`\n",
    "\n",
    "Note that when $\\textrm{bias}(\\hat{\\theta}_n) = 0$, the expectation of the estimator $\\hat{\\theta}_n$ is equal to the true value of parameter.  In this case, we say $\\hat{\\theta}_n$ is an unbiased estimator.  In general, an unbiased estimator is better than a biased estimator since its expectation is the same as the true parameter.\n",
    "\n",
    "\n",
    "It is worth being aware, however, that biased estimators are frequently used in practice.  There are cases where unbiased estimators do not exist without further assumptions, or are intractable to compute.  This may seem like a significant flaw in an estimator, however the majority of estimators encountered in practice are at least asymptotically unbiased in the sense that the bias tends to zero as the number of available samples tends to infinity: $\\lim_{n \\rightarrow \\infty} \\textrm{bias}(\\hat{\\theta}_n) = 0$.\n",
    "\n",
    "\n",
    "### Variance and Standard Deviation\n",
    "\n",
    "Second, let's measure the randomness in the estimator.  Recall from :numref:`sec_random_variables`, the *standard deviation* (or *standard error*) is defined as the squared root of the variance.  We may measure the degree of fluctuation of an estimator by measuring the standard deviation or variance of that estimator.\n",
    "\n",
    "$$\\sigma_{\\hat{\\theta}_n} = \\sqrt{\\textrm{Var} (\\hat{\\theta}_n )} = \\sqrt{E[(\\hat{\\theta}_n - E(\\hat{\\theta}_n))^2]}.$$\n",
    ":eqlabel:`eq_var_est`\n",
    "\n",
    "It is important to compare :eqref:`eq_var_est` to :eqref:`eq_mse_est`.  In this equation we do not compare to the true population value $\\theta$, but instead to $E(\\hat{\\theta}_n)$, the expected sample mean.  Thus we are not measuring how far the estimator tends to be from the true value, but instead we measuring the fluctuation of the estimator itself.\n",
    "\n",
    "\n",
    "### The Bias-Variance Trade-off\n",
    "\n",
    "It is intuitively clear that these two main components contribute to the mean squared error.  What is somewhat shocking is that we can show that this is actually a *decomposition* of the mean squared error into these two contributions plus a third one. That is to say that we can write the mean squared error as the sum of the square of the bias, the variance and the irreducible error.\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\textrm{MSE} (\\hat{\\theta}_n, \\theta) &= E[(\\hat{\\theta}_n - \\theta)^2] \\\\\n",
    " &= E[(\\hat{\\theta}_n)^2] + E[\\theta^2] - 2E[\\hat{\\theta}_n\\theta] \\\\\n",
    " &= \\textrm{Var} [\\hat{\\theta}_n] + E[\\hat{\\theta}_n]^2 + \\textrm{Var} [\\theta] + E[\\theta]^2 - 2E[\\hat{\\theta}_n]E[\\theta] \\\\\n",
    " &= (E[\\hat{\\theta}_n] - E[\\theta])^2 + \\textrm{Var} [\\hat{\\theta}_n] + \\textrm{Var} [\\theta] \\\\\n",
    " &= (E[\\hat{\\theta}_n - \\theta])^2 + \\textrm{Var} [\\hat{\\theta}_n] + \\textrm{Var} [\\theta] \\\\\n",
    " &= (\\textrm{bias} [\\hat{\\theta}_n])^2 + \\textrm{Var} (\\hat{\\theta}_n) + \\textrm{Var} [\\theta].\\\\\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "We refer the above formula as *bias-variance trade-off*. The mean squared error can be divided into three sources of error: the error from high bias, the error from high variance and the irreducible error. The bias error is commonly seen in a simple model (such as a linear regression model), which cannot extract high dimensional relations between the features and the outputs. If a model suffers from high bias error, we often say it is *underfitting* or lack of *flexibilty* as introduced in (:numref:`sec_generalization_basics`). The high variance usually results from a too complex model, which overfits the training data. As a result, an *overfitting* model is sensitive to small fluctuations in the data. If a model suffers from high variance, we often say it is *overfitting* and lack of *generalization* as introduced in (:numref:`sec_generalization_basics`). The irreducible error is the result from noise in the $\\theta$ itself.\n",
    "\n",
    "\n",
    "### Evaluating Estimators in Code\n",
    "\n",
    "Since the standard deviation of an estimator has been implementing by simply calling `a.std()` for a tensor `a`, we will skip it but implement the statistical bias and the mean squared error.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8f3f40cc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:38:58.073073Z",
     "iopub.status.busy": "2023-08-18T19:38:58.072247Z",
     "iopub.status.idle": "2023-08-18T19:38:58.077982Z",
     "shell.execute_reply": "2023-08-18T19:38:58.077011Z"
    },
    "origin_pos": 6,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "# Statistical bias\n",
    "def stat_bias(true_theta, est_theta):\n",
    "    return(torch.mean(est_theta) - true_theta)\n",
    "\n",
    "# Mean squared error\n",
    "def mse(data, true_theta):\n",
    "    return(torch.mean(torch.square(data - true_theta)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de7d958f",
   "metadata": {
    "origin_pos": 8
   },
   "source": [
    "To illustrate the equation of the bias-variance trade-off, let's simulate of normal distribution $\\mathcal{N}(\\theta, \\sigma^2)$ with $10,000$ samples. Here, we use a $\\theta = 1$ and $\\sigma = 4$. As the estimator is a function of the given samples, here we use the mean of the samples as an estimator for true $\\theta$ in this normal distribution $\\mathcal{N}(\\theta, \\sigma^2)$ .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b8e931ad",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:38:58.082116Z",
     "iopub.status.busy": "2023-08-18T19:38:58.081376Z",
     "iopub.status.idle": "2023-08-18T19:38:58.090486Z",
     "shell.execute_reply": "2023-08-18T19:38:58.089065Z"
    },
    "origin_pos": 10,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.0170)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta_true = 1\n",
    "sigma = 4\n",
    "sample_len = 10000\n",
    "samples = torch.normal(theta_true, sigma, size=(sample_len, 1))\n",
    "theta_est = torch.mean(samples)\n",
    "theta_est"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba979d43",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "Let's validate the trade-off equation by calculating the summation of the squared bias and the variance of our estimator. First, calculate the MSE of our estimator.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bca96ecd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:38:58.095807Z",
     "iopub.status.busy": "2023-08-18T19:38:58.095138Z",
     "iopub.status.idle": "2023-08-18T19:38:58.104592Z",
     "shell.execute_reply": "2023-08-18T19:38:58.103585Z"
    },
    "origin_pos": 13,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(16.0298)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mse(samples, theta_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bac2df74",
   "metadata": {
    "origin_pos": 14
   },
   "source": [
    "Next, we calculate $\\textrm{Var} (\\hat{\\theta}_n) + [\\textrm{bias} (\\hat{\\theta}_n)]^2$ as below. As you can see, the two values agree to numerical precision.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e88f47bb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:38:58.108277Z",
     "iopub.status.busy": "2023-08-18T19:38:58.107569Z",
     "iopub.status.idle": "2023-08-18T19:38:58.115012Z",
     "shell.execute_reply": "2023-08-18T19:38:58.114168Z"
    },
    "origin_pos": 16,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(16.0298)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bias = stat_bias(theta_true, theta_est)\n",
    "torch.square(samples.std(unbiased=False)) + torch.square(bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da17358d",
   "metadata": {
    "origin_pos": 18
   },
   "source": [
    "## Conducting Hypothesis Tests\n",
    "\n",
    "\n",
    "The most commonly encountered topic in statistical inference is hypothesis testing. While hypothesis testing was popularized in the early 20th century, the first use can be traced back to John Arbuthnot in the 1700s. John tracked 80-year birth records in London and concluded that more men were born than women each year. Following that, the modern significance testing is the intelligence heritage by Karl Pearson who invented $p$-value and Pearson's chi-squared test, William Gosset who is the father of Student's t-distribution, and Ronald Fisher who initialed the null hypothesis and the significance test.\n",
    "\n",
    "A *hypothesis test* is a way of evaluating some evidence against the default statement about a population. We refer the default statement as the *null hypothesis* $H_0$, which we try to reject using the observed data. Here, we use $H_0$ as a starting point for the statistical significance testing. The *alternative hypothesis* $H_A$ (or $H_1$) is a statement that is contrary to the null hypothesis. A null hypothesis is often stated in a declarative form which posits a relationship between variables. It should reflect the brief as explicit as possible, and be testable by statistics theory.\n",
    "\n",
    "Imagine you are a chemist. After spending thousands of hours in the lab, you develop a new medicine which can dramatically improve one's ability to understand math. To show its magic power, you need to test it. Naturally, you may need some volunteers to take the medicine and see whether it can help them learn mathematics better. How do you get started?\n",
    "\n",
    "First, you will need carefully random selected two groups of volunteers, so that there is no difference between their mathematical understanding ability measured by some metrics. The two groups are commonly referred to as the test group and the control group. The *test group* (or *treatment group*) is a group of individuals who will experience the medicine, while the *control group* represents the group of users who are set aside as a benchmark, i.e., identical environment setups except taking this medicine. In this way, the influence of all the variables are minimized, except the impact of the independent variable in the treatment.\n",
    "\n",
    "Second, after a period of taking the medicine, you will need to measure the two groups' mathematical understanding by the same metrics, such as letting the volunteers do the same tests after learning a new mathematical formula. Then, you can collect their performance and compare the results.  In this case, our null hypothesis will be that there is no difference between the two groups, and our alternate will be that there is.\n",
    "\n",
    "This is still not fully formal.  There are many details you have to think of carefully. For example, what is the suitable metrics to test their mathematical understanding ability? How many volunteers for your test so you can be confident to claim the effectiveness of your medicine? How long should you run the test? How do you decide if there is a difference between the two groups?  Do you care about the average performance only, or also the range of variation of the scores? And so on.\n",
    "\n",
    "In this way, hypothesis testing provides a framework for experimental design and reasoning about certainty in observed results.  If we can now show that the null hypothesis is very unlikely to be true, we may reject it with confidence.\n",
    "\n",
    "To complete the story of how to work with hypothesis testing, we need to now introduce some additional terminology and make some of our concepts above formal.\n",
    "\n",
    "\n",
    "### Statistical Significance\n",
    "\n",
    "The *statistical significance* measures the probability of erroneously rejecting the null hypothesis, $H_0$, when it should not be rejected, i.e.,\n",
    "\n",
    "$$ \\textrm{statistical significance }= 1 - \\alpha = 1 - P(\\textrm{reject } H_0 \\mid H_0 \\textrm{ is true} ).$$\n",
    "\n",
    "It is also referred to as the *type I error* or *false positive*. The $\\alpha$, is called as the *significance level* and its commonly used value is $5\\%$, i.e., $1-\\alpha = 95\\%$. The significance level can be explained as the level of risk that we are willing to take, when we reject a true null hypothesis.\n",
    "\n",
    ":numref:`fig_statistical_significance` shows the observations' values and probability of a given normal distribution in a two-sample hypothesis test. If the observation data example is located outsides the $95\\%$ threshold, it will be a very unlikely observation under the null hypothesis assumption. Hence, there might be something wrong with the null hypothesis and we will reject it.\n",
    "\n",
    "![Statistical significance.](../img/statistical-significance.svg)\n",
    ":label:`fig_statistical_significance`\n",
    "\n",
    "\n",
    "### Statistical Power\n",
    "\n",
    "The *statistical power* (or *sensitivity*) measures the probability of reject the null hypothesis, $H_0$, when it should be rejected, i.e.,\n",
    "\n",
    "$$ \\textrm{statistical power }= 1 - \\beta = 1 - P(\\textrm{ fail to reject } H_0  \\mid H_0 \\textrm{ is false} ).$$\n",
    "\n",
    "Recall that a *type I error* is error caused by rejecting the null hypothesis when it is true, whereas a *type II error* is resulted from failing to reject the null hypothesis when it is false. A type II error is usually denoted as $\\beta$, and hence the corresponding statistical power is $1-\\beta$.\n",
    "\n",
    "\n",
    "Intuitively, statistical power can be interpreted as how likely our test will detect a real discrepancy of some minimum magnitude at a desired statistical significance level. $80\\%$ is a commonly used statistical power threshold. The higher the statistical power, the more likely we are to detect true differences.\n",
    "\n",
    "One of the most common uses of statistical power is in determining the number of samples needed.  The probability you reject the null hypothesis when it is false depends on the degree to which it is false (known as the *effect size*) and the number of samples you have.  As you might expect, small effect sizes will require a very large number of samples to be detectable with high probability.  While beyond the scope of this brief appendix to derive in detail, as an example, want to be able to reject a null hypothesis that our sample came from a mean zero variance one Gaussian, and we believe that our sample's mean is actually close to one, we can do so with acceptable error rates with a sample size of only $8$.  However, if we think our sample population true mean is close to $0.01$, then we'd need a sample size of nearly $80000$ to detect the difference.\n",
    "\n",
    "We can imagine the power as a water filter. In this analogy, a high power hypothesis test is like a high quality water filtration system that will reduce harmful substances in the water as much as possible. On the other hand, a smaller discrepancy is like a low quality water filter, where some relative small substances may easily escape from the gaps. Similarly, if the statistical power is not of enough high power, then the test may not catch the smaller discrepancy.\n",
    "\n",
    "\n",
    "### Test Statistic\n",
    "\n",
    "A *test statistic* $T(x)$ is a scalar which summarizes some characteristic of the sample data.  The goal of defining such a statistic is that it should allow us to distinguish between different distributions and conduct our hypothesis test.  Thinking back to our chemist example, if we wish to show that one population performs better than the other, it could be reasonable to take the mean as the test statistic.  Different choices of test statistic can lead to statistical test with drastically different statistical power.\n",
    "\n",
    "Often, $T(X)$ (the distribution of the test statistic under our null hypothesis) will follow, at least approximately, a common probability distribution such as a normal distribution when considered under the null hypothesis. If we can derive explicitly such a distribution, and then measure our test statistic on our dataset, we can safely reject the null hypothesis if our statistic is far outside the range that we would expect.  Making this quantitative leads us to the notion of $p$-values.\n",
    "\n",
    "\n",
    "### $p$-value\n",
    "\n",
    "The $p$-value (or the *probability value*) is the probability that $T(X)$ is at least as extreme as the observed test statistic $T(x)$ assuming that the null hypothesis is *true*, i.e.,\n",
    "\n",
    "$$ p\\textrm{-value} = P_{H_0}(T(X) \\geq T(x)).$$\n",
    "\n",
    "If the $p$-value is smaller than or equal to a predefined and fixed statistical significance level $\\alpha$, we may reject the null hypothesis. Otherwise, we will conclude that we are lack of evidence to reject the null hypothesis. For a given population distribution, the *region of rejection* will be the interval contained of all the points which has a $p$-value smaller than the statistical significance level $\\alpha$.\n",
    "\n",
    "\n",
    "### One-side Test and Two-sided Test\n",
    "\n",
    "Normally there are two kinds of significance test: the one-sided test and the two-sided test. The *one-sided test* (or *one-tailed test*) is applicable when the null hypothesis and the alternative hypothesis only have one direction. For example, the null hypothesis may state that the true parameter $\\theta$ is less than or equal to a value $c$. The alternative hypothesis would be that $\\theta$ is greater than $c$. That is, the region of rejection is on only one side of the sampling distribution.  Contrary to the one-sided test, the *two-sided test* (or *two-tailed test*) is applicable when the region of rejection is on both sides of the sampling distribution. An example in this case may have a null hypothesis state that the true parameter $\\theta$ is equal to a value $c$. The alternative hypothesis would be that $\\theta$ is not equal to $c$.\n",
    "\n",
    "\n",
    "### General Steps of Hypothesis Testing\n",
    "\n",
    "After getting familiar with the above concepts, let's go through the general steps of hypothesis testing.\n",
    "\n",
    "1. State the question and establish a null hypotheses $H_0$.\n",
    "2. Set the statistical significance level $\\alpha$ and a statistical power ($1 - \\beta$).\n",
    "3. Obtain samples through experiments.  The number of samples needed will depend on the statistical power, and the expected effect size.\n",
    "4. Calculate the test statistic and the $p$-value.\n",
    "5. Make the decision to keep or reject the null hypothesis based on the $p$-value and the statistical significance level $\\alpha$.\n",
    "\n",
    "To conduct a hypothesis test, we start by defining a null hypothesis and a level of risk that we are willing to take. Then we calculate the test statistic of the sample, taking an extreme value of the test statistic as evidence against the null hypothesis. If the test statistic falls within the reject region, we may reject the null hypothesis in favor of the alternative.\n",
    "\n",
    "Hypothesis testing is applicable in a variety of scenarios such as the clinical trails and A/B testing.\n",
    "\n",
    "\n",
    "## Constructing Confidence Intervals\n",
    "\n",
    "\n",
    "When estimating the value of a parameter $\\theta$, point estimators like $\\hat \\theta$ are of limited utility since they contain no notion of uncertainty. Rather, it would be far better if we could produce an interval that would contain the true parameter $\\theta$ with high probability.  If you were interested in such ideas a century ago, then you would have been excited to read \"Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability\" by Jerzy Neyman :cite:`Neyman.1937`, who first introduced the concept of confidence interval in 1937.\n",
    "\n",
    "To be useful, a confidence interval should be as small as possible for a given degree of certainty. Let's see how to derive it.\n",
    "\n",
    "\n",
    "### Definition\n",
    "\n",
    "Mathematically, a *confidence interval* for the true parameter $\\theta$ is an interval $C_n$ that computed from the sample data such that\n",
    "\n",
    "$$P_{\\theta} (C_n \\ni \\theta) \\geq 1 - \\alpha, \\forall \\theta.$$\n",
    ":eqlabel:`eq_confidence`\n",
    "\n",
    "Here $\\alpha \\in (0, 1)$, and $1 - \\alpha$ is called the *confidence level* or *coverage* of the interval. This is the same $\\alpha$ as the significance level as we discussed about above.\n",
    "\n",
    "Note that :eqref:`eq_confidence` is about variable $C_n$, not about the fixed $\\theta$. To emphasize this, we write $P_{\\theta} (C_n \\ni \\theta)$ rather than $P_{\\theta} (\\theta \\in C_n)$.\n",
    "\n",
    "### Interpretation\n",
    "\n",
    "It is very tempting to interpret a $95\\%$ confidence interval as an interval where you can be $95\\%$ sure the true parameter lies, however this is sadly not true.  The true parameter is fixed, and it is the interval that is random.  Thus a better interpretation would be to say that if you generated a large number of confidence intervals by this procedure, $95\\%$ of the generated intervals would contain the true parameter.\n",
    "\n",
    "This may seem pedantic, but it can have real implications for the interpretation of the results.  In particular, we may satisfy :eqref:`eq_confidence` by constructing intervals that we are *almost certain* do not contain the true value, as long as we only do so rarely enough.  We close this section by providing three tempting but false statements.  An in-depth discussion of these points can be found in :citet:`Morey.Hoekstra.Rouder.ea.2016`.\n",
    "\n",
    "* **Fallacy 1**. Narrow confidence intervals mean we can estimate the parameter precisely.\n",
    "* **Fallacy 2**. The values inside the confidence interval are more likely to be the true value than those outside the interval.\n",
    "* **Fallacy 3**. The probability that a particular observed $95\\%$ confidence interval contains the true value is $95\\%$.\n",
    "\n",
    "Sufficed to say, confidence intervals are subtle objects.  However, if you keep the interpretation clear, they can be powerful tools.\n",
    "\n",
    "### A Gaussian Example\n",
    "\n",
    "Let's discuss the most classical example, the confidence interval for the mean of a Gaussian of unknown mean and variance.  Suppose we collect $n$ samples $\\{x_i\\}_{i=1}^n$ from our Gaussian $\\mathcal{N}(\\mu, \\sigma^2)$.  We can compute estimators for the mean and variance by taking\n",
    "\n",
    "$$\\hat\\mu_n = \\frac{1}{n}\\sum_{i=1}^n x_i \\;\\textrm{and}\\; \\hat\\sigma^2_n = \\frac{1}{n-1}\\sum_{i=1}^n (x_i - \\hat\\mu)^2.$$\n",
    "\n",
    "If we now consider the random variable\n",
    "\n",
    "$$\n",
    "T = \\frac{\\hat\\mu_n - \\mu}{\\hat\\sigma_n/\\sqrt{n}},\n",
    "$$\n",
    "\n",
    "we obtain a random variable following a well-known distribution called the *Student's t-distribution on* $n-1$ *degrees of freedom*.\n",
    "\n",
    "This distribution is very well studied, and it is known, for instance, that as $n\\rightarrow \\infty$, it is approximately a standard Gaussian, and thus by looking up values of the Gaussian c.d.f. in a table, we may conclude that the value of $T$ is in the interval $[-1.96, 1.96]$ at least $95\\%$ of the time.  For finite values of $n$, the interval needs to be somewhat larger, but are well known and precomputed in tables.\n",
    "\n",
    "Thus, we may conclude that for large $n$,\n",
    "\n",
    "$$\n",
    "P\\left(\\frac{\\hat\\mu_n - \\mu}{\\hat\\sigma_n/\\sqrt{n}} \\in [-1.96, 1.96]\\right) \\ge 0.95.\n",
    "$$\n",
    "\n",
    "Rearranging this by multiplying both sides by $\\hat\\sigma_n/\\sqrt{n}$ and then adding $\\hat\\mu_n$, we obtain\n",
    "\n",
    "$$\n",
    "P\\left(\\mu \\in \\left[\\hat\\mu_n - 1.96\\frac{\\hat\\sigma_n}{\\sqrt{n}}, \\hat\\mu_n + 1.96\\frac{\\hat\\sigma_n}{\\sqrt{n}}\\right]\\right) \\ge 0.95.\n",
    "$$\n",
    "\n",
    "Thus we know that we have found our $95\\%$ confidence interval:\n",
    "$$\\left[\\hat\\mu_n - 1.96\\frac{\\hat\\sigma_n}{\\sqrt{n}}, \\hat\\mu_n + 1.96\\frac{\\hat\\sigma_n}{\\sqrt{n}}\\right].$$\n",
    ":eqlabel:`eq_gauss_confidence`\n",
    "\n",
    "It is safe to say that :eqref:`eq_gauss_confidence` is one of the most used formula in statistics.  Let's close our discussion of statistics by implementing it.  For simplicity, we assume we are in the asymptotic regime.  Small values of $N$ should include the correct value of `t_star` obtained either programmatically or from a $t$-table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f55bb8e4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:38:58.119022Z",
     "iopub.status.busy": "2023-08-18T19:38:58.118680Z",
     "iopub.status.idle": "2023-08-18T19:38:58.133325Z",
     "shell.execute_reply": "2023-08-18T19:38:58.131422Z"
    },
    "origin_pos": 20,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(-0.0568), tensor(0.0704))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# PyTorch uses Bessel's correction by default, which means the use of ddof=1\n",
    "# instead of default ddof=0 in numpy. We can use unbiased=False to imitate\n",
    "# ddof=0.\n",
    "\n",
    "# Number of samples\n",
    "N = 1000\n",
    "\n",
    "# Sample dataset\n",
    "samples = torch.normal(0, 1, size=(N,))\n",
    "\n",
    "# Lookup Students's t-distribution c.d.f.\n",
    "t_star = 1.96\n",
    "\n",
    "# Construct interval\n",
    "mu_hat = torch.mean(samples)\n",
    "sigma_hat = samples.std(unbiased=True)\n",
    "(mu_hat - t_star*sigma_hat/torch.sqrt(torch.tensor(N, dtype=torch.float32)),\\\n",
    " mu_hat + t_star*sigma_hat/torch.sqrt(torch.tensor(N, dtype=torch.float32)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "378d4b12",
   "metadata": {
    "origin_pos": 22
   },
   "source": [
    "## Summary\n",
    "\n",
    "* Statistics focuses on inference problems, whereas deep learning emphasizes on making accurate predictions without explicitly programming and understanding.\n",
    "* There are three common statistics inference methods: evaluating and comparing estimators, conducting hypothesis tests, and constructing confidence intervals.\n",
    "* There are three most common estimators: statistical bias, standard deviation, and mean square error.\n",
    "* A confidence interval is an estimated range of a true population parameter that we can construct by given the samples.\n",
    "* Hypothesis testing is a way of evaluating some evidence against the default statement about a population.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Let $X_1, X_2, \\ldots, X_n \\overset{\\textrm{iid}}{\\sim} \\textrm{Unif}(0, \\theta)$, where \"iid\" stands for *independent and identically distributed*. Consider the following estimators of $\\theta$:\n",
    "$$\\hat{\\theta} = \\max \\{X_1, X_2, \\ldots, X_n \\};$$\n",
    "$$\\tilde{\\theta} = 2 \\bar{X_n} = \\frac{2}{n} \\sum_{i=1}^n X_i.$$\n",
    "    * Find the statistical bias, standard deviation, and mean square error of $\\hat{\\theta}.$\n",
    "    * Find the statistical bias, standard deviation, and mean square error of $\\tilde{\\theta}.$\n",
    "    * Which estimator is better?\n",
    "1. For our chemist example in introduction, can you derive the 5 steps to conduct a two-sided hypothesis testing? Given the statistical significance level $\\alpha = 0.05$ and the statistical power $1 - \\beta = 0.8$.\n",
    "1. Run the confidence interval code with $N=2$ and $\\alpha = 0.5$ for $100$ independently generated dataset, and plot the resulting intervals (in this case `t_star = 1.0`).  You will see several very short intervals which are very far from containing the true mean $0$.  Does this contradict the interpretation of the confidence interval?  Do you feel comfortable using short intervals to indicate high precision estimates?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bc138ce",
   "metadata": {
    "origin_pos": 24,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/1102)\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "required_libs": []
 },
 "nbformat": 4,
 "nbformat_minor": 5
}